Cross-modal image retrieval with deep mutual information maximization

In this paper, we study the cross-modal image retrieval, where inputs contain a source plus some text that describes certain modifications to and desired image. Prior work usually uses three-stage strategy tackle task: 1) extracting features of inputs; 2) fusing its modified obtain fusion feature; 3) learning similarity metric between via deep learning. Since classical image/text encoders can learn useful representations common pair-based loss functions distance are enough for people improve retrieval accuracy by designing new networks. However, these methods do not successfully handle modality gap caused inconsistent feature distributions different modalities, which greatly influences To alleviate problem, apply contrastive self-supervised method Deep InfoMax (DIM) [1] our approach bridge enhancing dependence text, image, their fusion. Specifically, narrows maximizing mutual information semantically representations. Moreover, seek an effective subspace consistent images utilizing low-level layer encoder high-level network. Extensive experiments on three large-scale benchmarks show have bridged modalities achieve state-of-the-art performance.